Spin Liquid Phases for Spin-1 systems on the Triangular lattice 
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Motivated by recent experiments on material Ba3NiSb209, we propose two novel spin liquid 
phases [A and B) for spin-1 systems on a triangular lattice. At the mean field level, both spin 
liquid phases have gapless fermionic spinon excitations with quadratic band touching, thus in both 
phases the spin susceptibility and 7 = Cv/T saturate to a constant at zero temperature, which 
»' ' are consistent with the experimental results on Ba3NiSb2 09. On the lattice scale, these spin liquid 

phases have Sp(4) ~ SO(5) gauge fluctuation; while in the long wavelength limit this Sp(4) gauge 
symmetry is broken down to U(l) x Z2 in type A spin liquid phase, and broken down to Z4, in type 
B phase. We also demonstrate that the A phase is the parent state of the ferro-quadrupole state, 
nematic state, and the noncoUinear spin density wave state. 
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A quantum spin liquid (QSL) is a ground state of an in- 
sulating magnet with vanishing static local moments and 
exotic emergent excitations. [l| Within spin wave theory 
for the simplest Heisenberg Hamiltonians, quantum fluc- 
tuations rapidly decrease with increasing spin quantum 
number S, so it is often believed that QSLs may occur 
only in the extreme case of S =1/2 spins. Indeed, the 
most promising empirical QSL materials are comprised 
of spin- 1/2 moments 0-0] • However, when the Hamilto- 
nian deviates from the Heisenberg form, quantum effects 
can be enhanced also for higher spin, leading to ground 
states beyond the usual magnetically ordered ones. The- 
oretically, biquadratic and other higher order exchange 
terms have been argued to favor multipolar ordered and 
QSL states in particular materials, such as the triangu- 
lar lattice spin-1 magnet NiGa2S4 [S- 12| and certain or- 
dered double perovskites[13j- Quite unexpectedly, recent 
experiments have evidenced QSL behavior in the spin-1 
magnet Ba3NiSb2 09, wit h sp ins residing on triangular 
lattices with AB stacking. [1J| Although the Curie- Weiss 
temperature of this material is 9cw ^ — 75K, no mag- 
netic ordering or phase transition was observed down to 
a temperature of Q.ibK , approximately 200 times lower 
than I^cvfI- The low temperature thermodynamics of 
this material is strikingly similar to that of the geomet- 
rically similar spin-1/2 organic triangular lattice QSLs 
[1, EE- III- In particular, the spin susceptibility x and lin- 
ear coefficient of specific heat 7 = c^/T in Ba3NiSb2 09 
both saturate to constants at low temperature 14 1. 

Most theoretical approaches to QSLs rely on slave par- 
ticle methods, and/or wave functions which correspond 
to slave particles. While these approaches have been ex- 
tensively developed for S =1/2 systems, there has been 
little theoretical work on them for the 5=1 case. We con- 
sider this here. To sharpen the discussion, we assume the 
presence of SU(2) spin symmetry, and seek QSL states 
in this framework which match the basic phenomenology 



so far observed in the low temperature thermodynamics. 

One way of studying spin-1 system is by introducing 
three flavors of fermionic spinon (a = 1 — 3) as follows 
ii[ll]: 5"* = flS^pfp, and S"^ are three spin-1 matri- 
ces. In order to guarantee the equivalence of the spin 
Hilbert space and the spinon Hilbert space, one must 
impose the gauge constraint fl afi,a = Ij fixing the 
spinon density locally to 1/3-filling. At the mean field 
level, the spinon forms a Fermi surface whose area is 
1/3 of the Brillouin zone. A spinon Fermi surface seems 
to be consistent with constant x and 7 observed experi- 
mentally. However, beyond the mean field theory, due to 
the single occupancy constraint, the spinon fermi surface 
is coupled to a dynamical U(l) gauge field. This U(l) 
gauge field has a "dressed" over-damped z — 3 dynam- 
ics due to its coupling to the Fermi surface, which leads 
to a 7 = Cy/T ~ temperature [20, 2l\, 

inconsistent with experiment. One solution of this prob- 
lem is to introduce pairing of the spinous in the mean 
field state. This has its own difficulties: either a gap 
is induced and impurities must be invoked to restore the 
proper thermodynamics, [22] or spin-rotational symmetry 
must be strongly broken. [I81] 

General Formalism 

We start instead by representing the spin-1 operators 
in the following way: 



Q,0=t,4.a=l,2 



(1) 



Here are three spin-1/2 Pauli matrices. Each spinon 
fa^a has two indices: a =t, i denotes spin and a = 1,2 
is an "orbital" quantum number. Thus we can consider 
not only the usual spin SU(2) rotations in the a — /3 
space, but also orbital SU(2) transformations in the a — h 
space. Matching with the spin Hilbert space requires not 
only constraining the total fermion number to half-filling 
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(two fermions per site), but also requiring each site to be 
an orbital SU(2) singlet, which guarantees that the spin 
space is a symmetric spin-1 representation: 



~ fa,a,ifa,a,. 
a=1.2 a=t,4, 

~ fa,a,i'''abfa;b,i — 



(2) 



Here r^j, are three Pauli matrices that operate on the 
orbital indices. A similar slave fermion formalism with 
orbital indices was introduced in Ref. [1^ , and it was ap- 
plied to two-orbital SU(A^) magnets that can be realized 
in Alkaline earth cold atoms |24l42a | . 

Due to these two independent constraints in Eq. [21 the 
spinon fa^a appears to have the following U(l) x SU(2) 
gauge symmetries: 

U ( 1 ) ^ • fa ,a,i ^ ^ fa ,a,i^ 

SU(2)„ : ^ W'^■^'\bfa,b,^■ 



(3) 



By rewriting fa,a,i in terms of Majorana fermions 7] as 
follows, however, a larger gauge symmetry is exposed: 



fa,a,i — (^a,a,l,z ~t- ^?7ct,a,2,i)- 



(4) 



On every site, rji has in total three two-component 
spaces, making the maximal possible transformation 
on r]i S0(8). Within this S0(8), the spin SU(2) 
transformations are generated by the three operators 
(<T^A^, cr^, cr^A^), where the Pauli matrices A" op- 
erate on the two-component space (Re[/], Im[/]). The 
total gauge symmetry on rj is the maximal subgroup 
of S0(8) that commutes with the spin-SU(2) operators. 
This is Sp(4) ^ S0(5) generated by the ten matrices 
T^ab = ^[Ta,Tb], where 



(5) 



These F° with a = 1 • • • 5 define five gamma matrices that 
satisfy the Clifford algebra {Fa,Fh} = 25ab- ^ab and F^ 
are all 8 x 8 hermitian matrices. Tab are all antisymmetric 
and imaginary, while Fq are symmetric. 

We consider a spin-1 Heisenberg model on the triangu- 
lar lattice with both nearest neighbor and 2nd neighbor 
antiferromagnetic couplings. Based on the above spinon 
representation of spin-1 operators, the Heisenberg model 
can be rewritten as follows: 

Jij^i^'j ^ 5^ "^ij fa,a,i'^apfP,a,ifjijjO'!^pfp^b,j 

-2j:y A*(, -jAbaji + Const, 



Decoupling through a hopping term is also possible, but 
we do not pursue this here. To analyze Eq. ([6]), we adopt 
a mean field ansatz with nonzero pairing (Aatji), so that 
the spinon fa^a fills a mean field band structure. To im- 
prove beyond mean field, a variational spin wave function 
may be obtained by projecting the mean field ground 
state to satisfy Eq. ([2]): 



IG.pin) = n = 2) P(ff = 0)1/,,,,). 



(7) 



The general formalism discussed above can describe 
many novel spin liquid states, with various different 
gauge fluctuations that are subgroups of Sp(4). Here 
we focus on simple states which satisfy the phenomenol- 
ogy of Ba3NiSb2 09[l3|, and in particular demand linear 
specific heat and constant susceptibility. We consider the 
following ansatz, which is a d+id pairing state of spinous: 



(Aa6,(M+e)) = [Sab^i"''^ +r^b^2 



(m) 



(e^ + iey)^, (8) 



where e is any of the nearest-neighbor or 2nd neighbor 
unit vectors, and A^™) with m = 1,2 denotes the pairing 
amplitude on the nearest and 2nd neighbor links respec- 
tively. This is a spin singlet but orbital triplet. Because 
the pair wave function vanishes when two spinous are on 
the same site, such states may be particularly insensitive 
to the projection in Eq. [71 

Continuum theory: In the majority of the paper, we 
consider the case with A2'"'' — 0. Then, expanded at 
k — 0, the low energy mean field Hamiltonian reads 

H ^ Tj'iidl - a2)Fi3 + 2a,a,F23}7?, 



13 



-a^A^ F23 = a«A^ 



(9) 



This mean field Hamiltonian has quadratic band- 
touching at fc = 0. Using the same method as intro- 
duced in Ref. 27|, one can verify that this mean field 



Hamiltonian breaks the Sp(4) gauge symmetry down to 
a U(l) X Z2 gauge symmetry: 



QtVti Qi e {1, F4}. 



(10) 



Notice that the U(l) and Z2 gauge transformations do 
not commute with each other. 

In addition to the quadratic band touching at fc = 0, 
depending on A'-™\ there are multiple Dirac points in 
the Brillouin zone. For instance, when A^^^ < A'^^^ 
there are Dirac points at the Brillouin zone corners 
Q = ±(47r/3, 0). A complex Dirac fermion field ip at 
momentum Q — (47r/3, 0) can be defined as 



^ab. ji — ^aP fa.a. j fp .b,'t 



(6) 



(11) 
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The low energy Hamiltonian for ip reads ~ 
ijj'^ [iVi'idx — i^23dy)4'- However, the Dirac fermion has 
vanishing density of states at zero energy, and thus con- 
tributes sub-dominantly to the rj spinon in many physical 
properties. 

The spinon carries a projective representation of phys- 
ical symmetry group. Under discrete symmetry transfor- 
mations, the low energy spinon fields 77 and "0 transform 
as: 
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i4ir/3 



(12) 



These transformations guarantee that there is no rele- 
vant fermion bilinear perturbation that does not break 
physical symmetry. For instance, the fermion bilinear 
fifi ^ V*^i2V breaks the time-reversal symmetry; thus 
it is forbidden in the Hamiltonian. 

Effect of gauge fluctuations: The spinous are coupled 
to a U(l) gauge field a^r45. The gauge field Lagrangian 
is renormalized by the fermion loop, which generates a 
mass gap for oq, thus uq can be ignored hereafter. The 
same fermion loop will also renormalize the dynamics of 
the transverse component of gauge field to be: 



+ — h Ci yUj^+ i;2g2 



uj,q 



e - 

uj.q 



e2 . 



l + C2e2 1og(3^) 



(13) 



In Lagrangian Ci, the first two terms come from the 
screening of spinous at the quadratic band touching, 
while the third term comes from the Dirac points. At 
low energy, the gauge field therefore obeys z — \ scal- 
ing with ~ g, so that the g^/e^ term is negligible in 
Eq. (fTS)) . For this reason, the gauge field decouples from 
•q (which has z = 2 scaling) at low energy, but remains 
strongly coupled to the z = \ Dirac fermion -0. 

Thermodynamic and transport properties: The finite 
density of states of the 77 spinon leads to a constant 7 — 
Cy/T at zero temperature. In terms of 77, the spin density 

is represented as 



(14) 



Since the spin density commutes with the mean field 
Hamiltonian Eq. I17[ turning on an external magnetic 



field creates a Fermi surface of 77 and ip, and since the 
density of states is finite at the quadratic band touching, 
the spin susceptibility saturates to a constant at zero 
temperature. Thus this spin liquid phase is consistent 
with the scaling of specific heat and spin susceptibility 
observed experimentally. 

We also find a unique scaling of thermal conductivity. 
The thermal conductivity is proportional to the specific 
heat and the velocity of entropy carriers of the system: 
K ~ cvl. Since the spinon has quadratic band touching, 
at low temperature the average velocity scales as 77 ~ 
T^/^, thus the thermal conductivity scales as k ~ T^^^. 
Note that the thermal conductivity contribution from the 
Dirac fermion and gauge field scales as T^, and is thus 
sub dominant. 

Fluctuating orders: The gauge invariant fermion bilin- 
ear operators can be viewed as physical order parame- 
ters with power-law correlations. They can be classified 
according to their transformations under symmetry and 
gauge symmetry. Some of the fermion bilinears are sum- 
marized as follows; 

(1) Spin density wave 

Sp = cos{Q ■ r)ni + sm{Q ■ f)n2, 
77i + 7772 ~ aiip'' Sip* + hirfSip^ 

(2) Spin — Nematic : 

d ^ a2ri'^Sr3ri + b2tp'' ST^-tp, 



(3) 



(4) 



Nematic : 

N ^^^Si ■ Si+e{ex + isy)"^ 

e 

N ^ Ni+ iN2 

a3(?/*ri377 -I- 77/*r2377) -f- bsi^^Tisip + iip'^T23ip); 
Spin — chirality : 



ijk£A 

C ^ a477*ri277 -I- b4ip^Ti2tp, 



(15) 



where S — (u^A^, cr^, a^A^) are the spin matrices. 

Several of these orders are germane to spin-one tri- 
angular antiferromagnets. The spin density wave or- 
der parameter is precisely that which describes the clas- 
sical 120° planar spin state, with Q — (47r/3,0) co- 
inciding with the Brillouin zone corner. As a conse- 
quence the spin structure factor of this state is singular 
at this momentum. Spin nematic order occurs naturally 
when biquadratic interactions are present in spin one 
systems. ll| In fact, d changes sign under the Z2 gauge 
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FIG. 1: a, The spin liquid we are considering contains a 
quadratic band touching at fc = (hexagon), and Dirac points 
(squares) at the corners of the Brillouin zone, b, with a 
nonzero and small nematic order A^i > 0, the quadratic band 
touching is split into two Dirac points, and the locations of 
the other Dirac points are shifted. 



transformation i] Tirj, so it is a headless nematic direc- 
tor. The physical order parameter is actually a bilinear 
of d, which corresponds to the ferro-quadrupoletensor 

^ o o 

Spatial nematic order, in which lattice rotation symmetry 
is broken but time-reversal and spin symmetry are pre- 
served, is described by TVi and N2- Order of this type was 
suggested for S=l triangular antiferromagnets in Ref.fo'], 
but also can be realized by spontaneous formation of Hal- 
dane chains. The spin-chirality order parameter C is less 
obvious from a microscopic perspective, but is a fluctu- 
ating order for this QSL state. 

At the mean field level, the equal time correlation 
of spin chirality, nematic, and spin density wave order 
parameters all fall off as \lr^\ the correlation of spin 
quadrupole order parameter falls off as 1/r*. The U(l) 
gauge fluctuation will modify the scaling dimension of the 
order parameters, and its correction can be calculated 
systematically using a 1/iV expansion. We will leave this 
calculation to future studies. 

Potential instabilities: One potential instability of this 
spin liquid state is instanton (monopole) proliferation of 
the compact U(l) gauge field. However, due to screen- 
ing by the gapless fermions, the instantons are greatly 
suppressed. By analogy with the theory of the algebraic 
spin liquid (28| (in which the z = 1 gauge field is similarly 
strongly coupled to Dirac fermions), we expect the spin 
liquid phase here to be similarly stable in principle. 

Furthermore, the mean field Hamiltonian Eq.[T7]is sub- 
ject to perturbations such as four-fermion interactions, 
which are marginal perturbations at the quadratic band 
touching according to naive power-counting. These four- 
fermion interactions can modify the correlation functions 
of the order parameters discussed above. The renormal- 
ization group may lead to weak run-away flow of these 
four-fermion interactions, which eventually can break the 



symmetry of the system, and develop one of the orders 
in Eq. [151 

If one of these orders develops, it can completely or 
partially gap the fermions and introduce interesting ef- 
fects. Nonzero spin nematic order, d ^ 0, gaps out the 
quadratic band touching and Dirac fermion ip- Depend- 
ing on the sign of 02 and &2 , a nonzero d drives the mean 
field band structure of spinon into either a quantum spin 
Hall type of topological insulator or a topologically trivial 
insulator. If the system is in a quantum spin Hall topo- 
logical insulator, assuming d is ordered along z direction, 
the quantized flux of U(l) gauge field a^r45 would carry 
spin S*^. Since the 2-|-Id photon phase of the U(l) gauge 
field is the condensate of gauge flux, the U(f ) spin rota- 
tion around z axis is spontaneously broken in the pho- 
ton phase, thus eventually the spin SU(2) symmetry is 
broken down to a discrete subgroup i.e. there are in to- 
tal three Goldstone modes instead of two. If the spinon 
band insulator has trivial topology, then the system is 
in an ordinary ferro-quadrupolar phase as discussed in 
Ref. dill. 

Weak spatial nematic order does not open a gap 
but only splits the quadratic band touching into Dirac 
fermions at two different momenta (Fig. [1} ; the original 
Dirac fermions ip also shift. When the nematic order 
magnitude is very strong, above some critical value, all 
the Dirac fermions meet and annihilate in pairs, and the 
spinous become fully gapped. 

Spin chirality order, which breaks time-reversal and 
reflection symmetries, gaps out both the quadratic band 
touching and the Dirac points. Depending on the sign 
of 04 and 64, a nonzero spin chirality order can drive the 
spinous into a topological Chern insulator, or a topolog- 
ically trivial band insulator with the same symmetry. In 
the former case, one obtains a chiral spin liquid, in which 
the U(l) gauge field a^r45 acquires a Chern-Simons term 
after integrating out the fermions. In the topologically 
trivial band insulator, the U(I) gauge field will become 
confined by instanton proliferation. 

Other phases: For 5 = 1 spins, we may also consider 
another state with A^'"'' and Aj'""* both nonzero, and 
jA^™'! ^ jAj™'*]. In this case, the spinous have two dif- 
ferent bands both with quadratic band touching at fc = 0, 
but they have different band curvature: 

H ^ r;*{(9^-a2)(^ri3 + Br25) 

+ 2a,9y(Ar23-sri5)}r?. (i7) 

A and B are two linear combinations of pairing ampli- 
tudes on nearest and 2nd neighbor links. In this state, 
the gauge symmetry is broken down to Z4: 

T]^ ^ Q^m. Q^ e {±1, ±r4}. (18) 

The Z4 gauge field has a deconfined phase in 2-1-1 dimen- 
sion, and this state is thus clearly locally stable. It also 
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exhibits non-zero finite spin susceptibility and 7 = C^jT 
at zero temperature. 

A similar d + id state with quadratic band touching 
can also be considered for spin- 1/2 systems on the tri- 
angular lattice. The same mean field Hamiltonian as 
Eq. [T7] applied, but without an orbital index. This state 
remains time-reversal and reflection invariant, and has 
Z2 gauge structure. One might consider this as a can- 
didate state for the spin liquids observed in the com- 
pounds K-(ET)2CWCN)3, EtMe3Sb[Pd(dmit)2]2, and 
BaaCuSbaOg 041 [11 ■ 

Future work: In Ref. [l^,!!^!, a variational Monte Carlo 
computation based on the Gutzwiller projected wave- 
function for 5* = 1/2 was used to compare the energy 
of various mean field spin liquid states. Extension of this 
method to our generalized 5 = 1 projected wavefunc- 
tion, Eq. [3 is a non-trivial and interesting problem for 
the future. 
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